Weighted resource cost matrix scheduler

ABSTRACT

A scheduler for a storage node uses multi-dimensional weighted resource cost matrices to schedule processing of IOs. A separate matrix is created for each computing node of the storage node via machine learning or regression analysis. Each matrix includes distinct dimensions for each emulation of the computing node for which the matrix is created. Each dimension includes modeled costs in terms of amounts of resources of various types required to process an IO of various IO types. An IO received from a host by a computing node is not scheduled for processing by that computing node unless enough resources are available at each emulation of that computing node. If enough resources are unavailable at an emulation, then the IO is forwarded to a different computing node that has enough resources at each of its emulations. A weighted resource cost for processing the IO is calculated and used to determine scheduling priority. The weights or regression coefficients from the model may be used to calculate weighted resource cost.

TECHNICAL FIELD

The subject matter of this disclosure is generally related to datastorage systems, and more particularly to scheduling the processing ofIOs (Input-Output operations) in a data storage system.

BACKGROUND

SANs (Storage Area Networks) are key building blocks of a data center.SANs provide servers known as “hosts” with block-level access to datathat is used by applications that run on the hosts. One type of SAN is astorage array that includes interconnected computing nodes that manageaccess to arrays of SSDs (solid-state drives) and HDDs (hard diskdrives). A single storage array may support multiple clusters of hosts,each of which may support multiple instances of an application.

IOs (input-output commands) received by the computing nodes of a SAN arebuffered and queued for processing by a scheduler. A typical schedulerfunctions based on queue depth. For example, a computing node with agreater number of queued IOs may be given preference over anothercomputing node with a lesser number of queued IOs.

SUMMARY

All examples, aspects and features mentioned in this document can becombined in any technically possible way.

In accordance with some implementations a method comprises: in a datastorage system comprising a plurality of computing nodes that receive IO(Input-Output operation) commands from a plurality of hosts, eachcomputing node comprising a plurality of different emulations,scheduling an IO received by a first computing node by: identifying aresource cost to process the received IO for each emulation of the firstcomputing node; calculating free resources available at each emulationof the first computing node; and scheduling processing of the receivedIO responsive to determining that the available free resources at eachemulation of the first computing node exceed the identified resourcecost for each corresponding emulation of the first computing node. Insome implementations identifying the resource cost to process thereceived IO for each emulation of the first computing node comprisesidentifying one or more of queue depth, processor utilization, localmemory utilization, shared memory utilization, bandwidth utilization ofFibre Channel hardware, iscsi hardware, nvme hardware, bandwidthutilization of internal InfiniBand hardware across different boards,compression IOs queue depth and compression hardware bandwidth,deduplication IOs queue depth and deduplication hardware bandwidth,encryption hardware bandwidth and queue depth, and memory bus bandwidth.In some implementations identifying the resource cost to process thereceived IO for each emulation of the first computing node comprisesidentifying the resource cost for each of a host adapter, a driveadapter, a remote replication adapter, a local replication and dataservices adapter, and an operating system. Some implementations comprisecalculating a weighted resource cost for processing the received IO.Some implementations comprise determining an order in which the receivedIO will be processed by the first computing node relative to otherscheduled IOs based on the weighted resource cost. Some implementationscomprise determining that available free resources at each emulation ofa second computing node exceed the identified resource cost for eachcorresponding emulation of the second computing node responsive todetermining that at least one of the available free resources at one ormore emulation of the first computing node does not exceed theidentified resource cost for each corresponding emulation of the firstcomputing node. Some implementations comprise determining that availablefree resources at each emulation of a third computing node exceed theidentified resource cost for each corresponding emulation of the thirdcomputing node responsive to determining that at least one of theavailable free resources at one or more emulation of the first computingnode does not exceed the identified resource cost for each correspondingemulation of the first computing node, and further comprising selectingthe second computing node responsive to the second computing node havinggreater available free resources than the third computing node. Someimplementations comprise calculating a weighted resource cost forprocessing the received IO. Some implementations comprise determining anorder in which the received IO will be processed by the second computingnode relative to other scheduled IOs based on the weighted resourcecost. Some implementations comprise adjusting resource allocation for anemulation of the first computing node based on recency, frequency, andresponse time of the IO.

In accordance with some implementations an apparatus comprises: aplurality of fabric-interconnected computing nodes that receive IO(Input-Output operation) commands from a plurality of hosts, eachcomputing node comprising a plurality of different emulations; aplurality of managed storage drives; and a scheduler that identifies,for each emulation of a first computing node of the plurality offabric-interconnected computing nodes, a resource cost to process afirst IO received by the first computing node from one of the hosts, thescheduler using matrices to calculate free resources available at eachemulation of the first computing node and scheduling processing of thefirst IO responsive to determining that the available free resources ateach emulation of the first computing node exceed the identifiedresource cost for each corresponding emulation of the first computingnode. In some implementations the matrices comprise one or more resourcetypes selected from the group consisting of: queue depth, processorutilization, local memory utilization, shared memory utilization,bandwidth utilization of Fibre Channel hardware, iscsi hardware, nvmehardware, bandwidth utilization of internal InfiniBand hardware acrossdifferent boards, compression IOs queue depth and compression hardwarebandwidth, deduplication IOs queue depth and deduplication hardwarebandwidth, encryption hardware bandwidth and queue depth, and memory busbandwidth. In some implementations the emulations of the first computingnode comprise a host adapter, a drive adapter, a remote replicationadapter, a local replication and data services adapter, and an operatingsystem. In some implementations the scheduler calculates a weightedresource cost for processing the first IO. In some implementations thescheduler determines an order in which the first IO will be processed bythe first computing node relative to other scheduled IOs based on theweighted resource cost. In some implementations the scheduler determineswhether other computing nodes have enough available free resources ateach emulation to process the first IO in response to determining thatat least one emulation of the first computing node lacks enoughavailable resources to process the first IO. In some implementations thescheduler selects one of the other computing nodes having enoughavailable free resources at each emulation to process the first IO basedon a comparison of amounts of available free resources. In someimplementations the scheduler adjusts resource allocation for anemulation of the first computing node based on recency, frequency, andresponse time of the IO.

In accordance with some implementations an apparatus comprises: aplurality of fabric-interconnected computing nodes that receive IO(Input-Output operation) commands from a plurality of hosts, eachcomputing node comprising a host adapter, a drive adapter, a remoteadapter, and an operating system; a plurality of managed storage drives;and a scheduler that: creates a multi-dimensional weighted resource costmatrix for each of the computing nodes, each of the cost matricescomprising a host adapter dimension, a drive adapter dimension, a localreplication and data services adapter dimension, a remote replicationadapter dimension, and an operating system dimension, each dimensioncomprising an indication of numbers of resources for each of a pluralityof resource types required for each of a plurality of IO types; andidentifies, for each emulation of a first computing node of theplurality of fabric-interconnected computing nodes, a resource cost toprocess a first IO received by the first computing node from one of thehosts, the scheduler using the multi-dimensional weighted resource costmatrices to calculate free resources available at each emulation of thefirst computing node and scheduling processing of the first IOresponsive to determining that the available free resources at eachemulation of the first computing node exceed the identified resourcecost for each corresponding emulation of the first computing node. Insome implementations the multi-dimensional weighted resource costmatrices comprise resource type weighting coefficients, and wherein thescheduler calculates a weighted resource cost for processing the firstIO and determines an order in which the first IO will be processed bythe first computing node relative to other scheduled IOs based on theweighted resource cost.

Although no advantages are necessarily associated with anyimplementations of the invention, some implementations mayadvantageously avoid scheduling IOs for processing by a computing nodethat lacks enough resources at one or more emulations to process theIOs. Previous systems, such as schedulers based solely on queue depth,typically consider only queue depth at one emulation. As a result, IOthreads could be stalled by lack of enough of a single resource type onanother emulation. Further, assigning processing priority based onweighted resource cost may inhibit highly weighted-costly IOs fromstalling less weighted-costly IOs.

Other aspects, features, and implementations may become apparent in viewof the detailed description and figures.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates a SAN node with a weighted resource cost matrixscheduler.

FIG. 2 illustrates generation of multi-dimensional weighted resourcecost matrices.

FIG. 3 illustrates operation of the weighted resource cost matrixscheduler.

FIG. 4 illustrates a multi-dimensional weighted resource cost matrix fora computing node.

FIG. 5 illustrates an existing resource matrix.

FIG. 6 illustrates a current workload cost matrix.

FIG. 7 illustrates adjustment of resource allocation.

DETAILED DESCRIPTION

Aspects of the inventive concepts will be described as being implementedin a data storage system that includes a host server and storage array.Such implementations should not be viewed as limiting. Those of ordinaryskill in the art will recognize that there are a wide variety ofimplementations of the inventive concepts in view of the teachings ofthe present disclosure. Some aspects, features, and implementationsdescribed herein may include machines such as computers, electroniccomponents, optical components, and processes such ascomputer-implemented procedures and steps. It will be apparent to thoseof ordinary skill in the art that the computer-implemented proceduresand steps may be stored as computer-executable instructions on anon-transitory computer-readable medium. Furthermore, it will beunderstood by those of ordinary skill in the art that thecomputer-executable instructions may be executed on a variety oftangible processor devices, i.e. physical hardware. For practicalreasons, not every step, device, and component that may be part of acomputer or data storage system is described herein. Those of ordinaryskill in the art will recognize such steps, devices, and components inview of the teachings of the present disclosure and the knowledgegenerally available to those of ordinary skill in the art. Thecorresponding machines and processes are therefore enabled and withinthe scope of the disclosure.

The terminology used in this disclosure is intended to be interpretedbroadly within the limits of subject matter eligibility. The terms“logical” and “virtual” are used to refer to features that areabstractions of other features, e.g. and without limitation abstractionsof tangible features. The term “physical” is used to refer to tangiblefeatures that possibly include, but are not limited to, electronichardware. For example, multiple virtual computers could operatesimultaneously on one physical computer. The term “logic” is used torefer to one or more of special purpose physical circuit elements,firmware, software, computer instructions that are stored on anon-transitory computer-readable medium and implemented by multi-purposetangible processors, including any combinations thereof.

FIG. 1 illustrates a SAN node 100 with a weighted resource cost matrixscheduler 105. The weighted resource cost matrix scheduler manages howIOs are scheduled for processing by the SAN node. The IOs can include,but are not limited to, READ, WRITE, WRITE SAME, UNMAP, Reservation,Inquiry, and vendor unique commands from a host server 150, 152 andanother SAN node 103 (or other type of remote storage system).

The SAN node 100, which may be referred to as a storage array, includesone or more bricks 102, 104. Each brick includes an engine 106 and oneor more DAEs (Drive Array Enclosures) 108, 110. Each DAE includesmanaged drives 101 of one or more technology types. Examples mayinclude, without limitation, SSDs such as flash and HDDs with spinningdisk storage media. Each DAE might include 24 or more managed drives butthe figure is simplified. Each engine 106 includes a pair ofinterconnected computing nodes 112, 114, which may be referred to as“storage directors.” Each computing node includes resources such as atleast one multi-core processor 116 and local memory 118. The processormay include CPUs (Central Processing Units), GPUs (Graphics ProcessingUnits), or both. The local memory 118 may include volatile RAM(Random-Access Memory) of any type, NVM (Non-Volatile Memory) such asSCM (Storage Class Memory), or both. Each computing node includes one ormore host adapters 120 for communicating with the hosts 150, 152. Eachhost adapter has resources for servicing IOs, e.g. processors, volatilememory, and ports via which the hosts may access the SAN node. Eachcomputing node also includes one or more disk adapters) 128 forcommunicating with the managed drives 101 in the DAEs 108, 110. Eachdisk adapter has resources for servicing IOs, e.g. processors, volatilememory, and ports via which the computing node may access the DAEs. Eachcomputing node may also include one or more CAs (Channel Adapters) 122for communicating with other computing nodes via an interconnectingfabric 124. Each computing node may also include one or more localreplication and data services adapters (LRDSA) 158, each of which mayhave resources for servicing IOs. The local data service adaptersperform data relocation, data pool management, memory pool management,CPU pool management, and storage tiering management. One or more OSs 107running on the SAN node has resources for servicing IOs and supports awide variety of other functions. Each computing node may allocate aportion or partition of its respective local memory 118 to a sharedmemory that can be accessed by other computing nodes, e.g. via DMA(Direct Memory Access) or RDMA (Remote DMA). The paired computing nodes112, 114 of each engine 106 provide failover protection and may bedirectly interconnected by communication links. An interconnectingfabric 130 enables implementation of an N-way active-active backend. Abackend connection group includes all DAs that can access the same driveor drives. In some implementations every drive adapter 128 in thestorage array can reach every DAE via the fabric 130. Further, in someimplementations every drive adapter in the SAN node can access everymanaged drive 101 in the SAN node. The weighted cost matrix schedulermay include program code stored in the memory 118 of the computing nodesand executed by the processors 116 of the computing nodes.

Data associated with applications 154, 156 running on the hosts 150, 152is maintained on the managed drives 101. The managed drives 101 are notdiscoverable by the hosts 150, 152 but the SAN node 100 creates alogical storage device 140 that can be discovered and accessed by thehosts. Without limitation, the logical storage device may be referred toas a production volume, production device, or production LUN, where LUN(Logical Unit Number) is a number used to identify the logical storagevolume in accordance with the SCSI (Small Computer System Interface)protocol. The logical storage device 140 is used by the hostapplications 154, 156 for storage of host application data. From theperspective of the hosts 150, 152 the logical storage device 140 is asingle drive having a set of contiguous fixed-size LBAs (logical blockaddresses) on which data used by instances of the host applicationresides. However, the host application data is stored at non-contiguousaddresses on various managed drives 101.

To service IOs from instances of a host application the SAN node 100maintains metadata that indicates, among various things, mappingsbetween LBAs of the logical storage device 140 and addresses with whichextents of host application data can be accessed from the shared memoryand managed drives 101. In response to a data access command from aninstance of one of the host applications to read data from theproduction volume 140 the SAN node uses the metadata to find therequested data in the shared memory or managed drives. When therequested data is already present in memory when the command is receivedit is considered a “cache hit.” When the requested data is not in theshared memory when the command is received it is considered a “cachemiss.” In the event of a cache miss the accessed data is temporarilycopied into the shared memory from the managed drives and used toservice the IO, i.e. reply to the host application with the data via oneof the computing nodes. In the case of a WRITE to the production volumethe SAN node copies the data into the shared memory, marks thecorresponding logical storage device location as dirty in the metadata,and creates new metadata that maps the logical storage device addresswith a location to which the data is eventually written on the manageddrives. READ and WRITE “hits” and “misses” occur depending on whetherthe stale data associated with the IO is present in the shared memorywhen the IO is received.

SAN node 103 maintains a replica (or snaps or backups or combinationsthereof) of the logical storage device 140. Updates to the logicalstorage device are shared between the SAN nodes. For example, if host150 sends a WRITE command to SAN node 100 that results in changes to thelogical storage device 140 then SAN node 100 communicates the changes toSAN node 103, e.g. with assistance from the OS. Each computing nodeincludes a remote replication adapter 121 for communicating with otherstorage systems. The remote replication adapter includes resources suchas processors, volatile memory, and ports for servicing IOs.

FIG. 2 illustrates a process for generation of multi-dimensionalweighted cost matrices. The steps may be implemented in whole or in partby the weighted cost matrix scheduler. Machine learning is used togenerate a model of the resource costs and response times and max IOPSfor processing different types of IOs as indicated in step 200. UsingIOs, resource utilization and IO response time and max IOPs as trainingdata, the machine learning technique models correlations betweendifferent IO types, response time, max IOPs and the resources used toservice those IO types. The model indicates costs in terms of thenumbers of resources of each individual resource type needed to servicea single IO of each type. Machine learning results in computation ofweights or regression coefficients that indicate strength of correlationbetween model parameters such as each resource type and response timeand maximum IOPs. The weights or regression coefficients are obtained bya decision tree-based ensemble machine learning algorithm that uses anextreme gradient boosting framework to generate non-linear regressionmodels. Each parameter in this model gets its individual weight duringtraining the extreme gradient boosting models. Each weight will not onlyrepresent corresponding resource weights but also all non-linearinteractions effects between resources within a computing node and allnon-linear interaction effects between resources across differentcomputing nodes. These important learned weights are used extensivelyduring distributed load balancing to pre compute the cost of an IO. Whenhardware or software resources are dynamically increased or decreasedthen those respective resource types weights are rebuilt using learnedmodels. The model is used to generate cost matrices as described below.

A multi-dimensional cost matrix is generated for each computing node asindicated in step 202. The multi-dimensional cost matrix includesseparate matrix dimensions for each resource-owning subsystem(emulation) of the SAN. For example, the host adapters, remotereplication adapters, disk adapters, local data services adapters, localreplication adapters, and OSs may each be considered emulations. Foreach dimension the corresponding matrix indicates the modeled resourcecosts for servicing each type of IO, the weights or regressioncoefficients for each resource type that were calculated during themodel training phase, and the numbers of each resource type that arecurrently available for use.

The multi-dimensional weighted cost matrices from each of the computingnodes are copied to the shared memory as indicated at step 204. The rateat which the local matrices are copied to shared memory is directlyproportional to the current workload on the storage array. Somepercentage of hardware resources are reserved exclusively for thisdistributed scheduling copy operation from local memory to sharedmemory, so that each adapter in the storage array gets the latest dataas soon as possible during peak storage array workload thus helpingbetter scheduling when it is most needed, thereby making the matricesavailable to the scheduler and all computing nodes. Steps 202 and 204are iterated over time to update the matrices based on current resourceusage, as will be discussed below.

FIG. 3 illustrates operation of the weighted cost matrix scheduler.After an IO is received at one of the computing nodes as indicated instep 300 the scheduler determines the modeled weighted resource cost ofthe IO for each resource type for each dimension/emulation of thatcomputing node as indicated at step 302. The modeled weighted resourcecosts may be determined from the multi-dimensional cost matrix for thecomputing node that received the IO. The modeled weighted costs are usedto determine whether to schedule the IO for processing. Only if thereare enough units of every resource type in every dimension of thatcomputing node to process the IO as determined from the cost matrix atstep 304 and if the computed cost is less compared to other alternativecost paths will the IO be approved for scheduling. It should be notedthat some resource types and some emulations may be uninvolved inprocessing of some types of IOs.

Having approved the IO for scheduling, a weighted cost is determined forthe IO as indicated in step 305. The weights or regression coefficientsfrom the model, which may be indicative of the value of each additionalresource of a given type in terms of response time and max IOPs, areused to calculate weighted cost. In some implementations the weightedcost is calculated by multiplying the number of required resources ofeach resource type by the weight or regression coefficient for thatresource type. The weighted costs of all required resource types may beaggregated into a single weighted cost for the IO. The IO is scheduledbased on weighted cost as indicated in step 306. More particularly, IOsthat are scheduled for processing based on resource costs beingsatisfiable may be scheduled to be processed in an order based onweighted cost. In some implementations IOs having a lower weighted costare scheduled ahead of IOs having relatively greater weighted resourcecost.

If it is determined at step 304 that there are insufficient resources atany one of the emulations/dimensions for the receiving computing node toprocess the IO, then the other multi-dimensional cost matrices in theshared memory are examined to determine whether another computing nodein the SAN has enough resources to process the IO as indicated at step308. If at least one other computing node has enough resources at allits emulations/dimensions, then another computing node is selected asindicated in step 310. If multiple computing nodes have enough resourcesto process the IO, then the computing node having the greatest amount offree resources may be selected. The weighted resource cost is thendetermined, and the IO is scheduled for processing by the selectedcomputing node as indicated at set 312, including scheduling based onweighted resource cost. If none of the computing nodes has enoughresources to process the IO, then the IO may be buffered, or a queuefull message may be returned to the host that sent the IO as indicatedin step 314. If the IO is buffered then flow returns to step 304,possibly after some time delay.

FIG. 4 illustrates a multi-dimensional weighted resource cost matrix 400for a computing node. The multi-dimensional weighted resource costmatrix 400 include multiple dimensions: a host adapter dimension 402, aremote adapter dimension 404, a disk adapter dimension 406, an OS(operating system) dimension 408, and a local data services andreplication adapter dimension 410. The host adapter, remote adapter,disk adapter, local data services and replication adapter, and OSemulations each have resources. For example, both the host adapter anddisk adapter have processors and volatile memory. To process an IO suchas a read or write it is necessary for all the host adapter, local dataservice adapter and disk adapter resources to have enough processor andmemory resources, otherwise the IO may stall. Consequently, distinctdimensions are created for each resource-owning sub-system. Eachemulation for each computing node may independently update itscorresponding cost matrix dimension and send the updated dimension tothe shared memory. Internal array fabric bandwidth, CPU cycles, messagebus, and other SAN resources may be explicitly reserved forscheduler-related traffic.

FIG. 5 illustrates an existing resource matrix 500, which is a componentof the multi-dimensional weighted resource cost matrix. An existingresource matrix includes values that represent total numbers of existingresource types in selected units for a single emulation of a singlecomputing node. Weights or regression coefficients are associated witheach of those resource types. For example, a computing node or SAN nodemay have 70,000 slots in the shared memory with a weight of “10” each,and 50,000 local buffers with a weight of “2” each. Thus, the weightedresource cost of one shared memory slot is 10 and the weighted resourcecost of one local buffer is 2, these weights are obtained by gradientboosted trees during non-linear modeling. Generally, the existingresource matrix does not change unless resource allocations within theSAN node are readjusted. For example, if the amount of shared memoryallocated for data is changed, and thus the number of slots is changed,then the shared memory write space in the existing resource matrix wouldbe updated accordingly.

FIG. 6 illustrates a current workload cost matrix 600, which is acomponent of the multi-dimensional weighted resource cost matrix. Thecurrent workload cost matrix includes values that represent the numbersof the existing resources that are currently being used to process IOs(and any other tasks) for a single emulation of a single computing node.Resource workloads included in the current workload cost matrix andexisting resource matrix may include queue depth, current CPUutilization, local memory utilization, shared memory utilization,bandwidth utilization of Fibre channel hardware, iscsi hardware, nvmehardware, bandwidth utilization of internal InfiniBand hardware acrossdifferent boards, compression IOs queue depth and compression hardwarebandwidth, deduplication IOs queue depth and deduplication hardwarebandwidth, encryption hardware bandwidth and queue depth and PCI memorybus or system memory bus bandwidth across CPUs. Queue depth is thenumber of requests vs responses per layer like Fiber Channel, NVME,driver layer, Host Adapter, cache layer, local data service, DiskAdapter, InfiniBand, message bus, compression, deduplication andencryption hardware and software queue depths. Queue depth acrossread-miss queue, read-miss response queue, write pyramid, Version WritePending, disk pool management queue, local replication queue, ReadPrefetch Q, Asynchronous write placement Q, scrubber, and low prioritytasks across all threads may be included, where the low priority tasksinclude each request and response across InfiniBand threads, local dataservice threads, local replication threads, storage tier threads,drivers threads for different type of storage hardware like FibreChannel, NVME, compression, deduplication, software compression,hardware compression, copy and replication threads, TCP/IP, fingerprint, life signs, configuration threads, operating system threads,cache management threads, message bus threads. Current CPU utilizationincludes number of CPUs, per core queue depth, latency, and utilization.The parameters may also include message bus queue depth, bandwidth andlatency, number of atomic machine compare and swap instructions viaInfiniBand hardware latency and queue depths, memory PCI utilization,and queue depth per different type of hardware and CPU cores.

The difference in corresponding numbers of resources between theexisting resource matrix and the current workload cost matrix for anemulation is the amount of free resources available from that emulationfor an IO being considered for scheduling. Referring to FIGS. 5 and 6,for example, if the IO command type is a READ then the modeled cost is 1atomic instruction access via InfiniBand hardware, 16 blocks ofbandwidth, 1 Direct Memory Access to shared memory etc. The freeresources available for that IO are 10,000−6000=4000 atomic instructionsaccess via InfiniBand, 16 GBPs−12 GBPs=4 GBPs blocks of bandwidth, 14GBPs−12 GBPs=2 GBPs direct memory access to shared memory etc. Becauseenough of all the resource types required for the READ are available,the READ IO is scheduled. The weighting coefficients are applied to thenumbers of resources needed to process the IO, e.g. as multipliers, tocalculate the weighted resource cost for processing the IO. For example,the weighted resource cost for using 10 local buffers is the number oflocal buffers required times the weight for a local buffer, i.e.10*2=20. The sum of all the weighted resource costs is the (total)weighted cots. The weighted resource cost may then be used to determinethe order in which the IO is scheduled for processing relative to otherscheduled IOs. For example, IOs having a lower weighted resource costmay be prioritized over IOs having a greater weighted resource cost.Consequently, a small number of weighted-costly IOs are less likely todelay processing of less weighted-costly IOs. This may result inlater-scheduled IOs being processed before earlier-scheduled IOs.

FIG. 7 illustrates adjustment of resource allocation. Each emulation mayindependently adjust resource allocation. Some other resources, e.g.shared memory allocations, are SAN-wide in scope. Recency 700, frequency702, and response time or cost 704 are used as inputs to determinewhether and how to adjust the resource allocations as indicated in step706. Recency 700 quantifies how recently each resource type was used andaverage response time for processing IOs during those recent uses.Frequency 702 quantifies how frequently each resource type is used oversome selected time period. Cost 704 includes the number of resources ofeach resource type required to process a type of IO command, like howmany scratch buffers are required, how much bandwidth is needed,percentage of CPU cycles or time, etc. If lack of an identified resourcetype is preventing scheduling or causing response time to fail to meet ahost quality of service target for recently received IOs with at least athreshold frequency then more of that resource type may be allocateddynamically, thereby increasing the existing resource. Adjustment ofresource allocation may include dynamically increasing and decreasinginternal software resources based on recency and frequency to align withincoming IO traffic.

Specific examples have been presented to provide context and conveyinventive concepts. The specific examples are not to be considered aslimiting. A wide variety of modifications may be made without departingfrom the scope of the inventive concepts described herein. Moreover, thefeatures, aspects, and implementations described herein may be combinedin any technically possible way. Accordingly, modifications andcombinations are within the scope of the following claims.

What is claimed is:
 1. A method comprising: in a data storage systemcomprising a plurality of computing nodes that receive IO (Input-Outputoperation) commands from a plurality of hosts, each computing nodecomprising a plurality of different emulations, scheduling an IOreceived by a first computing node by: identifying a resource cost toprocess the received IO for each emulation of the first computing node;calculating free resources available at each emulation of the firstcomputing node; and scheduling processing of the received IO responsiveto determining that the available free resources at each emulation ofthe first computing node exceed the identified resource cost for eachcorresponding emulation of the first computing node.
 2. The method ofclaim 1 wherein identifying the resource cost to process the received IOfor each emulation of the first computing node comprises identifying oneor more of queue depth, processor utilization, local memory utilization,shared memory utilization, bandwidth utilization of Fibre Channelhardware, iscsi hardware, nvme hardware, bandwidth utilization ofinternal InfiniBand hardware across different boards, compression IOsqueue depth and compression hardware bandwidth, deduplication IOs queuedepth and deduplication hardware bandwidth, encryption hardwarebandwidth and queue depth and memory bus bandwidth.
 3. The method ofclaim 1 wherein identifying the resource cost to process the received IOfor each emulation of the first computing node comprises identifying theresource cost for each of a host adapter, a drive adapter, a remotereplication adapter, a local replication and data services adapter, andan operating system.
 4. The method of claim 1 comprising calculating aweighted resource cost for processing the received IO.
 5. The method ofclaim 4 comprising determining an order in which the received IO will beprocessed by the first computing node relative to other scheduled IOsbased on the weighted resource cost.
 6. The method of claim 1 comprisingdetermining that available free resources at each emulation of a secondcomputing node exceed the identified resource cost for eachcorresponding emulation of the second computing node responsive todetermining that at least one of the available free resources at one ormore emulation of the first computing node does not exceed theidentified resource cost for each corresponding emulation of the firstcomputing node.
 7. The method of claim 6 comprising determining thatavailable free resources at each emulation of a third computing nodeexceed the identified resource cost for each corresponding emulation ofthe third computing node responsive to determining that at least one ofthe available free resources at one or more emulation of the firstcomputing node does not exceed the identified resource cost for eachcorresponding emulation of the first computing node, and furthercomprising selecting the second computing node responsive to the secondcomputing node having greater available free resources than the thirdcomputing node.
 8. The method of claim 7 comprising calculating aweighted resource cost for processing the received IO.
 9. The method ofclaim 8 comprising determining an order in which the received IO will beprocessed by the second computing node relative to other scheduled IOsbased on the weighted resource cost.
 10. The method of claim 1comprising adjusting resource allocation for an emulation of the firstcomputing node based on recency, frequency, and response time.
 11. Anapparatus comprising: a plurality of fabric-interconnected computingnodes that receive IO (Input-Output operation) commands from a pluralityof hosts, each computing node comprising a plurality of differentemulations; a plurality of managed storage drives; and a scheduler thatidentifies, for each emulation of a first computing node of theplurality of fabric-interconnected computing nodes, a resource cost toprocess a first IO received by the first computing node from one of thehosts, the scheduler using matrices to calculate free resourcesavailable at each emulation of the first computing node and schedulingprocessing of the first IO responsive to determining that the availablefree resources at each emulation of the first computing node exceed theidentified resource cost for each corresponding emulation of the firstcomputing node.
 12. The apparatus of claim 11 wherein the matricescomprise one or more resource types selected from the group consistingof: IO queue depth, processor utilization, local memory utilization,shared memory utilization, bandwidth utilization of Fibre Channelhardware, iscsi hardware, nvme hardware, bandwidth utilization ofinternal InfiniBand hardware across different boards, compression IOsqueue depth and compression hardware bandwidth, deduplication IOs queuedepth and deduplication hardware bandwidth, encryption hardwarebandwidth and queue depth and memory bus bandwidth.
 13. The apparatus ofclaim 11 wherein the emulations of the first computing node comprise ahost adapter, a drive adapter, a remote replication adapter, a localreplication and data service adapter, and an operating system.
 14. Theapparatus of claim 11 wherein the scheduler calculates a weightedresource cost for processing the first IO.
 15. The apparatus of claim 14wherein the scheduler determines an order in which the first IO will beprocessed by the first computing node relative to other scheduled IOsbased on the weighted resource cost.
 16. The apparatus of claim 11wherein the scheduler determines whether other computing nodes haveenough available free resources at each emulation to process the firstIO in response to determining that at least one emulation of the firstcomputing node lacks enough available resources to process the first IO.17. The apparatus of claim 16 wherein the scheduler selects one of theother computing nodes having enough available free resources at eachemulation to process the first IO based on a comparison of amounts ofavailable free resources.
 18. The apparatus of claim 11 wherein thescheduler adjusts resource allocation for an emulation of the firstcomputing node based on recency, frequency, and response time.
 19. Anapparatus comprising: a plurality of fabric-interconnected computingnodes that receive IO (Input-Output operation) commands from a pluralityof hosts, each computing node comprising a host adapter, a driveadapter, a remote adapter, and an operating system; a plurality ofmanaged storage drives; and a scheduler that: creates amulti-dimensional weighted resource cost matrix for each of thecomputing nodes, each of the cost matrices comprising a host adapterdimension, a drive adapter dimension, a remote adapter dimension, and anoperating system dimension, each dimension comprising an indication ofnumbers of resources for each of a plurality of resource types requiredfor each of a plurality of IO types; and identifies, for each emulationof a first computing node of the plurality of fabric-interconnectedcomputing nodes, a resource cost to process a first IO received by thefirst computing node from one of the hosts, the scheduler using themulti-dimensional weighted resource cost matrices to calculate freeresources available at each emulation of the first computing node andscheduling processing of the first IO responsive to determining that theavailable free resources at each emulation of the first computing nodeexceed the identified resource cost for each corresponding emulation ofthe first computing node.
 20. The apparatus of claim 19 wherein themulti-dimensional weighted resource cost matrices comprise resource typeweighting coefficients, and wherein the scheduler calculates a weightedresource cost for processing the first IO and determines an order inwhich the first IO will be processed by the first computing noderelative to other scheduled IOs based on the weighted resource cost.